[SPARK-13184][SQL] Add a datasource-specific option minPartitions in HadoopFsRelation#options #13320

maropu · 2016-05-26T07:06:39Z

What changes were proposed in this pull request?

This pr adds a new option minPartitions accepted in HadoopFsRelation#options.
This is a datasource-specific value of the minimal splitting number for input data.

How was this patch tested?

Added unit tests in FileSourceStrategySuite.

rxin · 2016-05-26T07:15:43Z

I was thinking actually just use the same config key name, and override it. Can you read my comment on the jira ticket and let me know if that's clear?

maropu · 2016-05-26T07:16:56Z

oh, sorry. I'll check and fix it.

SparkQA · 2016-05-26T08:32:04Z

Test build #59353 has finished for PR 13320 at commit dbee9b3.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2016-05-26T16:01:46Z

@rxin Could you check this again to satisfy your intention?
Also, I'll make the description up-to-date later.

SparkQA · 2016-05-26T17:24:34Z

Test build #59388 has finished for PR 13320 at commit e703f48.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

rxin · 2016-05-28T04:19:18Z

I'm going to mark this for 2.1 given we are too close to 2.0 to modify behaviors like this.

maropu · 2016-05-29T07:02:48Z

Jenkins, retest this please.

SparkQA · 2016-05-29T08:34:26Z

Test build #59586 has finished for PR 13320 at commit e703f48.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-06-30T19:56:37Z

Test build #61555 has finished for PR 13320 at commit e28586b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2016-06-30T20:20:04Z

sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/WriterContainer.scala

@@ -67,9 +67,6 @@ private[sql] abstract class BaseWriterContainer(
  // This is only used on driver side.
  @transient private val jobContext: JobContext = job

-  private val speculationEnabled: Boolean =
-    relation.sparkSession.sparkContext.conf.getBoolean("spark.speculation", defaultValue = false)
-


Deleted because this line not used.

SparkQA · 2016-06-30T22:02:05Z

Test build #61566 has finished for PR 13320 at commit c77bd36.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-01T00:35:42Z

Test build #61577 has finished for PR 13320 at commit abc0054.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-07-18T18:38:34Z

Test build #62473 has finished for PR 13320 at commit abc0054.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-08-19T08:55:02Z

Test build #64052 has finished for PR 13320 at commit 6440370.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2016-08-19T18:48:16Z

Test build #64077 has finished for PR 13320 at commit ef06b82.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2016-08-22T14:18:20Z

@rxin could you give me comments?

SparkQA · 2016-11-18T18:54:07Z

Test build #68863 has finished for PR 13320 at commit ba3f2a0.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

maropu · 2017-03-03T06:05:22Z

@gatorsmile Could you check this and give me comments, too?

SparkQA · 2017-06-12T14:18:21Z

Test build #77932 has finished for PR 13320 at commit ba3f2a0.

This patch fails Spark unit tests.
This patch does not merge cleanly.
This patch adds no public classes.

gatorsmile · 2017-06-12T20:29:26Z

sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala

+  // Returns all option set in the `SparkConf`, the `SQLConf`, and a given data source `options`.
+  // If the same keys exist, they are overridden with ones in the `options`.
+  private def optionsOverriddenWith(options: Map[String, String]): Map[String, String] = {
+    sparkSession.sparkContext.conf.getAllAsMap ++ sparkSession.conf.getAll ++ options


Here, we just cover the code path for DataFrameReader APIs.

gatorsmile · 2017-06-12T20:30:24Z

To fundamentally resolve this issue, we need to consider all the read/write paths. A lot of works to do for achieving it. Do you want to continue the work or just close it?

maropu · 2017-06-13T00:28:31Z

Ah, ok. I'll check again.

HyukjinKwon · 2017-07-24T03:35:54Z

@maropu, I just wonder if it is in progress in any way.

maropu · 2017-07-24T03:57:37Z

No, so I'll close this for now and we move this discussion to jira.

### What changes were proposed in this pull request? This pr aims to upgrade netty from 4.1.92 to 4.1.93. ### Why are the changes needed? 1.v4.1.92 VS v4.1.93 netty/netty@netty-4.1.92.Final...netty-4.1.93.Final 2.The new version brings some bug fix, eg: - Reset byte buffer in loop for AbstractDiskHttpData.setContent ([#13320](netty/netty#13320)) - OpenSSL MAX_CERTIFICATE_LIST_BYTES option supported ([#13365](netty/netty#13365)) - Adapt to DirectByteBuffer constructor in Java 21 ([#13366](netty/netty#13366)) - HTTP/2 encoder: allow HEADER_TABLE_SIZE greater than Integer.MAX_VALUE ([#13368](netty/netty#13368)) - Upgrade to latest netty-tcnative to fix memory leak ([#13375](netty/netty#13375)) - H2/H2C server stream channels deactivated while write still in progress ([#13388](netty/netty#13388)) - Channel#bytesBefore(un)writable off by 1 ([#13389](netty/netty#13389)) - HTTP/2 should forward shutdown user events to active streams ([#13394](netty/netty#13394)) - Respect the number of bytes read per datagram when using recvmmsg ([#13399](netty/netty#13399)) 3.The release notes as follows: - https://netty.io/news/2023/05/25/4-1-93-Final.html 4.Why not upgrade to `4-1-94-Final` version? Because the return value of the 'threadCache()' (from `PoolThreadCache` to `PoolArenasCache`) method of the netty Inner class used in the 'arrow memory netty' version '12.0.1' has changed and belongs to break change, let's wait for the upgrade of the 'arrow memory netty' before upgrading to the '4-1-94-Final' version. The reference is as follows: https://github.com/apache/arrow/blob/6af660f48472b8b45a5e01b7136b9b040b185eb1/java/memory/memory-netty/src/main/java/io/netty/buffer/PooledByteBufAllocatorL.java#L164 https://github.com/netty/netty/blob/da1a448d5bc4f36cc1744db93fcaf64e198db2bd/buffer/src/main/java/io/netty/buffer/PooledByteBufAllocator.java#L732-L736 ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Pass GA. Closes #41681 from panbingkun/upgrade_netty. Authored-by: panbingkun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>

maropu force-pushed the SPARK-13184 branch from e703f48 to e28586b Compare June 30, 2016 18:18

maropu reviewed Jun 30, 2016
View reviewed changes

maropu force-pushed the SPARK-13184 branch from c77bd36 to abc0054 Compare June 30, 2016 22:23

maropu force-pushed the SPARK-13184 branch from abc0054 to 6440370 Compare August 19, 2016 07:11

maropu added 3 commits November 19, 2016 01:23

Add a datasource-specific option 'minPartitions'

00b9757

Override the options of SparkConf/SQLConf with ones for datasources

ad5121f

Override all the options in datasources

debd8ed

maropu force-pushed the SPARK-13184 branch from ef06b82 to 022e0fd Compare November 18, 2016 16:27

Fix a bug

ba3f2a0

maropu force-pushed the SPARK-13184 branch from 022e0fd to ba3f2a0 Compare November 18, 2016 16:28

gatorsmile reviewed Jun 12, 2017

View reviewed changes

maropu closed this Jul 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-13184][SQL] Add a datasource-specific option minPartitions in HadoopFsRelation#options #13320

[SPARK-13184][SQL] Add a datasource-specific option minPartitions in HadoopFsRelation#options #13320

maropu commented May 26, 2016

rxin commented May 26, 2016

maropu commented May 26, 2016

SparkQA commented May 26, 2016

maropu commented May 26, 2016

SparkQA commented May 26, 2016

rxin commented May 28, 2016

maropu commented May 29, 2016

SparkQA commented May 29, 2016

SparkQA commented Jun 30, 2016

maropu Jun 30, 2016

SparkQA commented Jun 30, 2016

SparkQA commented Jul 1, 2016

SparkQA commented Jul 18, 2016

SparkQA commented Aug 19, 2016

SparkQA commented Aug 19, 2016

maropu commented Aug 22, 2016

SparkQA commented Nov 18, 2016

maropu commented Mar 3, 2017

SparkQA commented Jun 12, 2017

gatorsmile Jun 12, 2017

gatorsmile commented Jun 12, 2017

maropu commented Jun 13, 2017

HyukjinKwon commented Jul 24, 2017

maropu commented Jul 24, 2017

[SPARK-13184][SQL] Add a datasource-specific option minPartitions in HadoopFsRelation#options #13320

[SPARK-13184][SQL] Add a datasource-specific option minPartitions in HadoopFsRelation#options #13320

Conversation

maropu commented May 26, 2016

What changes were proposed in this pull request?

How was this patch tested?

rxin commented May 26, 2016

maropu commented May 26, 2016

SparkQA commented May 26, 2016

maropu commented May 26, 2016

SparkQA commented May 26, 2016

rxin commented May 28, 2016

maropu commented May 29, 2016

SparkQA commented May 29, 2016

SparkQA commented Jun 30, 2016

maropu Jun 30, 2016

Choose a reason for hiding this comment

SparkQA commented Jun 30, 2016

SparkQA commented Jul 1, 2016

SparkQA commented Jul 18, 2016

SparkQA commented Aug 19, 2016

SparkQA commented Aug 19, 2016

maropu commented Aug 22, 2016

SparkQA commented Nov 18, 2016

maropu commented Mar 3, 2017

SparkQA commented Jun 12, 2017

gatorsmile Jun 12, 2017

Choose a reason for hiding this comment

gatorsmile commented Jun 12, 2017

maropu commented Jun 13, 2017

HyukjinKwon commented Jul 24, 2017

maropu commented Jul 24, 2017